fix: azure-app-onboard & prereq - Harden app-onboard family: naming, schemas, cleanup - #2970
Open
vaibbavis wants to merge 7 commits into
Open
fix: azure-app-onboard & prereq - Harden app-onboard family: naming, schemas, cleanup#2970vaibbavis wants to merge 7 commits into
vaibbavis wants to merge 7 commits into
Conversation
vaibbavis
changed the title
JasonYeMSFT (JasonYeMSFT)
marked this pull request as ready for review
July 28, 2026 20:36
JasonYeMSFT (JasonYeMSFT)
requested review from
a team,
Rick Winter (RickWinter) and
Tom Meschter (tmeschter)
as code owners
July 28, 2026 20:36
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens and aligns the azure-app-onboard skill family’s phase contracts (session schemas, artifact expectations, naming, validation/manifests) and updates deploy guidance + Vally eval termination behavior to be more deterministic.
Changes:
- Tightens session/artifact schemas and documentation (e.g., required per-component
verdicts, added tenant/user identity fields, phase enum trimming). - Makes scaffold/deploy more contract-driven (plan-as-source-of-truth naming, standardized validation result shape, deploy skeleton/portal link + redeploy behavior).
- Updates Vally eval suites to early-terminate on artifact writes rather than ambiguous assistant text.
Show a summary per file
| File | Description |
|---|---|
| plugins/azure-skills/skills/azure-app-onboard/SKILL.md | Orchestrator step table hardening and updated MCP-unavailable disclaimer targets. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/SKILL.md | Scaffold phase guidance updates (backup behavior, troubleshooting additions). |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/validation-and-manifest.md | Manifest/validation check schema normalization and explicit manifest write path guidance. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/subagent-validate.md | Conformance validation guidance updated to enforce plan-verbatim naming. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/subagent-iac-gen.md | IaC generation guidance tightened (deployerObjectId resolution expectations). |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/self-review-checklist.md | Removes scaffold-stage what-if; clarifies validation responsibilities. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/scaffold-schemas.ts | Conformance result schema tightened (source always "script"), doc wording updates. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/env-var-secrets.md | Flexible Server admin username mapping clarified. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/bicep-patterns.md | Naming guidance changed to “plan is source of truth,” with fallback guidance. |
| plugins/azure-skills/skills/azure-app-onboard/scaffold/references/bicep-patterns-security.md | Clarifies deployer RBAC source (deployerObjectId). |
| plugins/azure-skills/skills/azure-app-onboard/references/session-schemas.ts | Session schema hardened (required component verdicts, new identity/staleness fields, phase enum trim). |
| plugins/azure-skills/skills/azure-app-onboard/references/session-protocol.md | Session artifact location table updated (and backup-path guidance removed). |
| plugins/azure-skills/skills/azure-app-onboard/references/pipeline-rules.md | Adjusts pipeline rule timing (intent write step). |
| plugins/azure-skills/skills/azure-app-onboard/references/pipeline-rules-runtime.md | Updates runtime guidance for secrets with shell-special chars. |
| plugins/azure-skills/skills/azure-app-onboard/references/handoff-protocol.md | Step number alignment with orchestrator. |
| plugins/azure-skills/skills/azure-app-onboard/references/azd-template-routing.md | “Start fresh” backup behavior clarified; avoids replaced-files path behavior. |
| plugins/azure-skills/skills/azure-app-onboard/references/approval-gates.md | Deploy approval ordering clarified and backup behavior aligned with azd-template routing. |
| plugins/azure-skills/skills/azure-app-onboard/prepare/SKILL.md | Prepare phase step ordering revised; adds reference pointer for MCP tool parameters. |
| plugins/azure-skills/skills/azure-app-onboard/prepare/references/prepare-schemas.ts | Schema docs aligned with new prepare step numbering; naming suffix field added. |
| plugins/azure-skills/skills/azure-app-onboard/deploy/SKILL.md | Deploy instructions hardened (portal link regen on redeploy, status messaging guardrails, error capture guidance). |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/subagent-preflight.md | Updates deploy-result skeleton shape (duration nesting). |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/preflight-checks.md | Preflight name-collision sync-back and what-if formatting changes. |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/error-classification.md | Healing guidance aligned to plan-verbatim naming (plan name updates). |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/deploy-schemas.ts | Deploy schema adjustments (completedUtc optional, remove unused status). |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/deploy-safety.md | Tightens “sanctioned secret file” and variable-passing guidance. |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/deploy-checklist-template.md | Adds explicit “warnings are not a stop signal” rules and status gating. |
| plugins/azure-skills/skills/azure-app-onboard/deploy/references/blocked-patterns.md | Refines secret-handling blocks (literal vs variable; sanctioned cache file). |
| plugins/azure-skills/skills/azure-app-onboard-prereq/SKILL.md | Strengthens mandatory-read language and cloud SDK gate phrasing. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/subscription-resolution.md | Adds login attempt cap + clearer halt behavior. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/session-schemas.ts | Prereq copy of shared session schema updated to match orchestrator copy. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/session-protocol.md | Azure login gate updated (timeouts, single-write merge incl. displayName, .gitignore enforcement). |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/remediation-protocol.md | Clarifies “no lockfile regeneration” as part of static-only remediation. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/readiness-gate.md | Tightens semantics of overallHealth vs axis verdicts; updates user-choice phrasing. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/prereq-schemas.ts | Adds Cloud SDK findings schema for migration gating. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/prereq-artifacts.md | Clarifies verdicts persistence contract and enum separation. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/deployability-check.md | Reclassifies external managed-service deps as WARN (mapping concern), not blockers. |
| plugins/azure-skills/skills/azure-app-onboard-prereq/references/completeness-check.md | Fixes fixPhase spelling (post-deploy). |
| evals/azure-app-onboard/prepare.eval.yaml | Early-terminate rules updated to stop on artifact writes rather than assistant text. |
| evals/azure-app-onboard/onboard.eval.yaml | Early-terminate rules refined and adds artifact-based remediation proof checks. |
| evals/azure-app-onboard-prereq/eval.yaml | Terminator regex adjusted and readiness-report existence checks repositioned. |
Review details
- Files reviewed: 40/40 changed files
- Comments generated: 3
- Review effort level: Low
JasonYeMSFT (JasonYeMSFT)
previously approved these changes
Jul 29, 2026
JasonYeMSFT (JasonYeMSFT)
approved these changes
Jul 29, 2026
JasonYeMSFT (JasonYeMSFT)
requested changes
Jul 29, 2026
vaibbavisk20
requested review from
kunalsuri-microsoft and
vaibbavisk20
as code owners
August 13, 2026 17:51
JasonYeMSFT (JasonYeMSFT)
previously approved these changes
Aug 19, 2026
Member
|
vaibbavis The changes look good to me but I am not sure why there are merge conflicts. Please resolve them and let me know when it's done. |
vaibbavisk20
requested review from
Madhura Bharadwaj (MadhuraBharadwaj-MSFT),
Xiaofu Huang (XiaofuHuang),
anchenyi,
Ankit Sinha (ankitbko),
chandraneel,
Tom Gamble (gambtho),
Justin Gonzales (jugonzales),
Paul Yuknewicz (paulyuk),
rakal-dyh,
Sai Koumudi Kaluvakolanu (saikoumudi),
Christopher T Earley (tendau) and
vebudumu
as code owners
August 20, 2026 18:29
vaibbavisk20
requested review from
Alex Thompson (alex-thompson),
joybb and
Kashif Khan (kashifkhan)
as code owners
August 20, 2026 18:29
…update eval suites and prereq references - Add deploy-conformance.sh/.ps1 — runtime conformance checks for deploy sub-skill - Add scaffold-conformance.sh/.ps1 — runtime conformance checks for scaffold sub-skill - Add code-deployment-appservice.md reference to deploy skill - Restore portal-links.md (was deleted, has portal deep-link patterns) - Update prereq eval suite (prereq/eval.yaml): revised stimulus/grader coverage - Update scaffold/seeded-deploy/e2e eval suites: grader and scoring adjustments - Update prereq readiness-gate.md and completeness-check.md - Update deploy preflight-checks.md and deploy-checklist-template.md
vaibbavisk20
force-pushed
the
apponboard-hardening-v2
branch
from
August 20, 2026 18:36
2c83b37 to
cf4660d
Compare
added 3 commits
August 20, 2026 11:37
- subagent-iac-gen.md: document appPort sourced from buildRequirements.exposedPort ?? 8080 for Container Apps - bicep-container-apps.md: add param appPort int = 8080 declaration (was used but undeclared) - prereq-schemas.ts: correct estimatedInstallTime comment (scaffold hardcodes 1800, field is informational) - deploy/SKILL.md: Steps 6 and 9 now reference portal-links.md as canonical source - preflight-checks.md: az deployment group what-if (consistent with sub-scope pattern) - iac-generation-rules.md: Step 10a main thread runs format, not validate subagent - deploy-strategy.md: buildRequirements is top-level on PrereqOutput, not per-component
JasonYeMSFT (JasonYeMSFT)
previously approved these changes
Aug 20, 2026
| 1. Any finding with `verdict: "FAIL"` → axis verdict MUST be `"FAIL"`. | ||
| 2. Any finding with `verdict: "WARN"` + `fixPhase: "prereq"` → escalate to `"FAIL"` (prevents wasting a deploy cycle). ⛔ Escalate only WARNs that would actually break THIS deploy (build/startup failure, or a health probe wired to a route the app lacks). Issues that deploy and run fine — missing trust proxy, README, in-memory sessions — stay `fixPhase: "postdeploy"`/`"scaffold"`; `engines`/health-endpoint escalate only on a real version/probe mismatch (see [completeness-check.md](completeness-check.md) § Stack-Specific Checks). | ||
| ⛔ **Verdict propagation cross-check** before computing `overallHealth` (statuses below are `verdicts` axes — see the enum note above): | ||
| 1. Any axis finding → that component's `verdicts.{axis}` MUST be `"FAIL"`. |
There was a problem hiding this comment.
This turns every finding, including a non-blocking WARN, into FAIL. That contradicts the next rule and the overallHealth mapping above, so an informational warning blocks the pipeline. Restore the verdict check here so only FAIL findings force the axis to fail.
Suggested change
| 1. Any axis finding → that component's `verdicts.{axis}` MUST be `"FAIL"`. | |
| 1. Any finding with `verdict: "FAIL"` -> that component's `verdicts.{axis}` MUST be `"FAIL"`. |
| > ⛔ `fixPhase: "prereq"` is a remediation trigger regardless of severity. If left unfixed → deploy-time failures. Warnings with `fixPhase: "scaffold"/"deploy-gate"/"post-deploy"` are NOT included. | ||
| 2. After fixes, ⛔ **re-run full Step 3 evaluation (all 3 axes)** on affected components. Re-read each reference file and re-evaluate inline. | ||
| 3. ⛔ **Verify fixes via static analysis only.** File exists → exports match imports (grep) → no syntax errors → config values present. Do NOT run install/build/test commands. | ||
| 3. ⛔ **Verify fixes via static analysis only.** File exists → exports match imports (grep) → no syntax errors → config values present. Do NOT run install/build/test commands. ⛔ **This includes lockfile regeneration:** editing a manifest (`package.json`, `pyproject.toml`, `go.mod`) is a static fix, but running `npm`/`pnpm`/`bun install`, `uv lock`, `pip install`, or `go mod tidy` to update the lockfile is a **build operation deferred to the deploy build** (Oryx / `az acr build` reconciles the lockfile there). NEVER regenerate a lockfile during remediation — leave the manifest edited and let the consented deploy build reconcile it. |
There was a problem hiding this comment.
This leaves dependency manifests and lockfiles inconsistent, then assumes the deploy build will repair them. Generated Node Dockerfiles use npm ci, which exits on a lockfile mismatch, and frozen-lockfile workflows fail the same way. Either regenerate the lockfile after explicit build consent or do not edit the dependency manifest. We should not mark remediation complete with a stale lockfile.
|
|
||
| 1. Write override to `context.json.overrides[]`: `{ "key": "ignoreExistingInfra", "value": "true", "reason": "User chose greenfield over existing azd template" }` | ||
| 2. If `infra/` directory exists, rename it to `infra.bak/` (single folder rename). This preserves the user's existing IaC as a backup before scaffold writes new files. | ||
| 2. If `infra/` directory exists, rename it to `infra.bak/` (single folder rename). If `azure.yaml` exists, rename it to `azure.yaml.bak`. This preserves the user's existing IaC as a backup before scaffold writes new files, and clears the azd-template detection signal (`azure.yaml` + `infra/`) so future sessions don't re-flag the repo as existing IaC. Tell the user their originals are preserved at these `.bak` paths. |
There was a problem hiding this comment.
These fixed .bak names are not collision safe. On a second Start fresh run, the directory rename can fail or nest under the old backup, and azure.yaml.bak can overwrite the user's prior backup. Use a session-scoped or timestamped backup path and refuse to overwrite an existing target before renaming.
| | `Compress-Archive` path flattening | PowerShell's `Compress-Archive -Path $files.FullName` uses absolute paths, flattening directory structure | Use `System.IO.Compression.ZipFile` with relative paths instead | | ||
| | AADSTS530084 (Terraform) | Token protection conditional access policy breaks `azurerm` provider auth; regular `az` CLI commands work fine | Re-scaffold as Bicep | | ||
| | Secret values with shell special chars | Passwords containing `$`, `` ` ``, `!`, `'`, `"` break when passed as inline CLI args (`--parameters key=val`) | **ALWAYS pass secrets via `main.parameters.json` or `terraform.tfvars`** — never as inline `--parameters` args. For deploy-phase secret seeding (`az keyvault secret set`), use `--file` with a temp file or pipe from stdin to avoid shell interpolation. | | ||
| | Secret values with shell special chars | Passwords containing `$`, `` ` ``, `!`, `'`, `"` break when passed as inline CLI args (`--parameters key=val`) | Generate URL-safe (`openssl rand -base64 32 \| tr -d '/+='`), reload from the git-ignored session `deploy-secrets.env`, and pass via a **quoted shell variable** (`--parameters key="$var"`) — never a literal. | |
There was a problem hiding this comment.
Quoting a shell variable prevents interpolation bugs, but the expanded secret is still present in the child process command line and can be exposed through process inspection. This also regresses the previous file or stdin guidance. Keep the secret out of argv by using a protected parameter file, stdin, or a service-supported environment variable.
| 1. Any finding with `verdict: "FAIL"` → axis verdict MUST be `"FAIL"`. | ||
| 2. Any finding with `verdict: "WARN"` + `fixPhase: "prereq"` → escalate to `"FAIL"` (prevents wasting a deploy cycle). ⛔ Escalate only WARNs that would actually break THIS deploy (build/startup failure, or a health probe wired to a route the app lacks). Issues that deploy and run fine — missing trust proxy, README, in-memory sessions — stay `fixPhase: "postdeploy"`/`"scaffold"`; `engines`/health-endpoint escalate only on a real version/probe mismatch (see [completeness-check.md](completeness-check.md) § Stack-Specific Checks). | ||
| ⛔ **Verdict propagation cross-check** before computing `overallHealth` (statuses below are `verdicts` axes — see the enum note above): | ||
| 1. Any axis finding → that component's `verdicts.{axis}` MUST be `"FAIL"`. |
There was a problem hiding this comment.
This makes every finding, including a WARN, force the axis to FAIL. That contradicts the next rule and collapses readyWithCaveats into blocked for ordinary warnings. Keep the propagation conditional on a FAIL finding.
Suggested change
| 1. Any axis finding → that component's `verdicts.{axis}` MUST be `"FAIL"`. | |
| 1. Any finding with `verdict: "FAIL"` → that component's `verdicts.{axis}` MUST be `"FAIL"`. |
| 1. Any finding with `verdict: "FAIL"` → axis verdict MUST be `"FAIL"`. | ||
| 2. Any finding with `verdict: "WARN"` + `fixPhase: "prereq"` → escalate to `"FAIL"` (prevents wasting a deploy cycle). ⛔ Escalate only WARNs that would actually break THIS deploy (build/startup failure, or a health probe wired to a route the app lacks). Issues that deploy and run fine — missing trust proxy, README, in-memory sessions — stay `fixPhase: "postdeploy"`/`"scaffold"`; `engines`/health-endpoint escalate only on a real version/probe mismatch (see [completeness-check.md](completeness-check.md) § Stack-Specific Checks). | ||
| ⛔ **Verdict propagation cross-check** before computing `overallHealth` (statuses below are `verdicts` axes — see the enum note above): | ||
| 1. Any axis finding → that component's `verdicts.{axis}` MUST be `"FAIL"`. |
There was a problem hiding this comment.
Any axis finding also includes WARN findings, so this forces every warning to FAIL before line 42 can apply its narrower escalation rule. That turns deployable apps with informational warnings into overallHealth: "blocked". Let's restore the severity check here so only findings with verdict: "FAIL" propagate a failed axis.
| ## ⛔ Read deploy/SKILL.md | ||
| - You MUST `view` deploy/SKILL.md BEFORE running any `az deployment` command | ||
| - Path: `plugin/skills/azure-app-onboard/deploy/SKILL.md` | ||
| - Path: `../SKILL.md` |
There was a problem hiding this comment.
This template is copied into .copilot-azure/sessions/{id}/deploy-checklist.md, so ../SKILL.md resolves under .copilot-azure/sessions/, not to the deploy skill. After compaction, the agent cannot reload the mandatory deploy workflow from this instruction. Fill this placeholder with the resolved deploy skill path when generating the checklist.
| 1. `infra/bicepconfig.json` — write `{ "formatting": { "newlineKind": "LF" } }` if it doesn't already exist (user's repo may have one). LF is critical because Bicep triple-quoted strings pass content literally to ARM, and `\r` bytes crash `/bin/sh` in containers. | ||
| 2. `infra/main.bicep` — subscription scope, RG creation with tags, module calls for all services + `role-assignments` module (KV deployer + app-to-KV RBAC), all unconditional. | ||
| 3. `infra/main.parameters.json` — ARM JSON format (NOT `.bicepparam`). Include `environmentName`, `location`, `sessionId`, `deployedBy`, `createdAt`. ⛔ **`createdAt` value:** run `Get-Date -Format "o"` in terminal to get the current ISO 8601 timestamp — NEVER use a hardcoded or placeholder date. Do NOT include `@secure()` params (passed at deploy time). Include `deployerObjectId` param (deploy phase passes via `az ad signed-in-user show --query id -o tsv`). | ||
| 3. `infra/main.parameters.json` — ARM JSON format (NOT `.bicepparam`). Include `environmentName`, `location`, `sessionId`, `deployedBy`, `createdAt`. ⛔ **`createdAt` value:** run `Get-Date -Format "o"` in terminal to get the current ISO 8601 timestamp — NEVER use a hardcoded or placeholder date. Do NOT include `@secure()` params (passed at deploy time). Include `deployerObjectId` param — ⛔ **resolve the REAL value now** via `az ad signed-in-user show --query id -o tsv` and write it into `main.parameters.json` (scaffold and deploy run as the same signed-in user). NEVER leave it empty or a placeholder — an empty `deployerObjectId` makes the deployer Key Vault role assignment invalid and causes a 403 when deploy seeds KV secrets. |
There was a problem hiding this comment.
Resumable sessions do not guarantee that scaffold and deploy use the same signed-in principal. Persisting the scaffold-time object ID grants Key Vault Secrets Officer to the old account, while the current deployer cannot seed secrets and receives a 403. Resolve and pass the current deployerObjectId during each deploy or what-if instead of baking it into main.parameters.json here.
| > ⛔ `fixPhase: "prereq"` is a remediation trigger regardless of severity. If left unfixed → deploy-time failures. Warnings with `fixPhase: "scaffold"/"deploy-gate"/"post-deploy"` are NOT included. | ||
| 2. After fixes, ⛔ **re-run full Step 3 evaluation (all 3 axes)** on affected components. Re-read each reference file and re-evaluate inline. | ||
| 3. ⛔ **Verify fixes via static analysis only.** File exists → exports match imports (grep) → no syntax errors → config values present. Do NOT run install/build/test commands. | ||
| 3. ⛔ **Verify fixes via static analysis only.** File exists → exports match imports (grep) → no syntax errors → config values present. Do NOT run install/build/test commands. ⛔ **This includes lockfile regeneration:** editing a manifest (`package.json`, `pyproject.toml`, `go.mod`) is a static fix, but running `npm`/`pnpm`/`bun install`, `uv lock`, `pip install`, or `go mod tidy` to update the lockfile is a **build operation deferred to the deploy build** (Oryx / `az acr build` reconciles the lockfile there). NEVER regenerate a lockfile during remediation — leave the manifest edited and let the consented deploy build reconcile it. |
There was a problem hiding this comment.
Leaving a changed manifest paired with a stale lockfile makes the generated Node path fail deterministically because dockerfile-generation.md requires npm ci --omit=dev, and npm ci rejects mismatched package.json and package-lock.json. The deploy build does not reconcile that lockfile first. Either regenerate the lockfile with explicit consent or leave this dependency remediation unresolved until the lockfile can be synchronized.
| - **Azure IaC** (`.bicep`, `azure.yaml`, `.tf` with `azurerm`): `ask_user` → "Start fresh" (rename `infra/` → `infra.bak/` and `azure.yaml` → `azure.yaml.bak`, then tell the user their originals are preserved at those `.bak` paths) or "Use existing" (route to `azure-prepare`, stop pipeline). | ||
| - **Non-Azure IaC** (`.tf` with GCP/AWS): respect `context.json.overrides[].iacFormat` from prepare. Default: Bicep alongside existing TF. | ||
| - **Unknown TF** (`detectedInfraProvider.terraform` == `"unknown"`): ask user which provider before routing. | ||
| - **No IaC**: continue. |
There was a problem hiding this comment.
The dispatched task runs from the user's repository workspace, not this skill directory, so ../deploy/references/deploy-checklist-template.md resolves outside the repository and the checklist generator cannot read its template. Pass the resolved skill-root path into the task prompt instead of relying on a relative working directory.
…kup collision, secret in argv, deployerObjectId, relative paths
vaibbavisk20
force-pushed
the
apponboard-hardening-v2
branch
from
August 21, 2026 17:21
4deea59 to
13bdb37
Compare
added 2 commits
August 21, 2026 10:23
…opagation, lockfile consent-before-edit, secret params file, deployerObjectId at deploy time, collision-safe backups, resolved skill paths
…pproval gate cleanup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Harden app-onboard family: naming, schemas, cleanup
Consistency/hardening pass across the app-onboard pipeline — no new features. It tightens the contracts between phases so downstream steps read stable data, and removes dead reference files. Grouped by phase for review.
Cross-cutting — session schema (
session-schemas.ts, both prereq + orchestrator copies)verdictson each component is now required (was optional) and documented as separate from the top-leveloverallHealthenum — prepare/readiness scoring depend on it.tenantId+userDisplayName(deploy identity/tags),lastScanCommit(prereq staleness guard).AppOnboardPhaseto the 5 real phases; expandedrouteToSkill/routeReasondocs.Prereq
deployability-check.md: apps using SQLite/Postgres/Redis/Kafka that map to an Azure service now classifyreadyWithCaveats, notblocked— they're a mapping concern, not a blocker. (behavior change — worth a close look.)prereq-schemas.ts: newCloudSdkFindingtype +cloudSdkFindings[]so the cloud-SDK gate can surface AWS/GCP deps and route toazure-cloud-migrate.prereq-artifacts.md: per-componentverdictsmust persist; clarified it's a plural object, not singular.completeness-check.md:fixPhasespellingpostdeploy→post-deploy; SKILL.md mandatory-read phrasing hardened.Prepare
prepare-schemas.ts);intentnow written after Step 4 (was Step 2).Scaffold
bicep-patterns.md,subagent-validate.md): scaffold usesprepare-plan.json.naming.resources[]verbatim instead of re-deriving withuniqueString()/take()/substring(). Why: re-deriving produced names that didn't match the plan and failed the conformance gate.self-review-checklist.md: noaz deployment sub what-ifat scaffold (needs real secrets) — deploy runs it.{ passed: true };ConformanceResult.sourcealways"script"; manifest written to the exact session path.env-var-secrets.md: use bareadministratorLogin(Flexible Server rejects the legacy{admin}@{server}form).cicd-pipelines.md,error-handling.md,self-review-procedure.md,waf-checklist.md.Deploy
SKILL.md: portal-link generation inlined into Step 6 (added a Terraform activity-log variant; re-emits a fresh link on redeploy). Failures recorded intoresourceResults[].error+healingAttempts[].preflight-checks.md: rename-on-collision writes the accepted name back into the plan before redeploy (keeps scaffold/deploy in sync); dropped--what-if-result-format FullResourcePayloads.deploy-schemas.ts:completedUtcoptional (set at finalize);ResourceDeployStatusdrops unusedskipped.portal-links.md(moved intoSKILL.md).Orchestrator
approval-gates.md: after deploy approval, readdeploy-checklist.md(compaction-safe anchor) first, thendeploy/SKILL.md; existingazure.yaml→ renameinfra/toinfra.bak/.SKILL.md: MCP-unavailable disclaimer now targets the specificprepare-plan.json.assumptions(scaffold gate) andcostEstimate.disclaimer(deploy gate) fields.Evals & tooling
onboard.eval.yaml/prepare.eval.yaml: early-stop matchers now fire on theprepare-plan.json/scaffold-manifest.jsonwrite-result (not spoken text); added acloudSdkFindingsgrader.run-onboard-vally.prompt.mdtweaks + logged the 07-22/07-23 runs (README index).Validation — eval run 2026-07-27 (vally v0.7,
claude-sonnet-4.6, real Azure for deploy)Combined: 51/57 tests · 448/464 graders. All 8 suites clear the 80% weighted threshold.
3 of 5 real deploys reached
status:"succeeded"+healthStatus:"healthy"with live endpoints. 0 harness errors.Why seeded-deploy is 80.8% (vs 96–100% elsewhere): it's 2 stimuli — App Service Pipeline 100% (25/25) and Container Apps Pipeline 61.5% (16/26). The Container Apps half early-stopped at the approval gate and never ran
az acr build/az deployment, so it lost the 10 downstream graders (succeeded status, live endpoint, healthy, deployment-summary.md, cleanup). Averaging a perfect run with one early-stop pulls the suite to 80.8% — it's one flaky stop, not a systemic seeded-deploy problem (no RG was even created).All 6 failures are known recurring flaky patterns, not regressions from this PR: 1 approval-gate early-stop (above) · 2 missing
scaffold-manifest.json(async-flush; one deploy still succeeded) · 3 prereq non-determinism (overallHealthclassification, missingreadiness-report.md+ axis verdicts,routeToSkillnot persisted).The new schema fields are exercised and passing where the model persists them (Cloud SDK Continue 7/7, Go Gin routing 5/5) — the 3 prereq misses are occasional non-persistence, not schema errors.
Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:tests/,npm run test:integration -- <skill>ornpm run test:vally -- --skill <skill>)Related Issues